ci: add import profiler check across monorepo by hebaalazzeh · Pull Request #17657 · googleapis/google-cloud-python

hebaalazzeh · 2026-07-08T02:41:02Z

Overview

This PR integrates the import profiler script (introduced in #17467) into our automated CI pipeline.

Why this matters: Import times significantly impact CLI responsiveness and cold starts for Serverless products like Cloud Run and Cloud Functions. Ideally, library imports should stay under 500ms, and anything taking over 1 second is a target for optimization. This CI check helps us proactively track metrics like import latency, memory footprint, and code volume to prevent performance regressions on critical libraries.

The primary goal of this check is to track and enforce performance standards for package import times across the repository, especially following our recent work on lazy loading and cold-start optimizations.

By running this benchmark as a CI check with a defined dynamic differential failure threshold, we can programmatically prevent latency regressions in module initialization times before they are merged, ensuring downstream consumers aren't impacted by unexpectedly slow startup times.

Changes Included

GitHub Actions Workflow: Created a new workflow (.github/workflows/import-profiler.yml) that triggers on PRs and merge groups. The workflow is pinned specifically to Python 3.15.
Native CI Integration (No template bloat): Rather than polluting the central gapic-generator template (noxfile.py.j2) and forcing updates across 150+ packages, the CI script (ci/run_single_test.sh) natively handles the profiler. It automatically spins up a lightweight virtual environment, installs the target package, and runs the profiler.
Dynamic Differential Checks: Enhanced ci/run_single_test.sh to checkout HEAD^1 (the main branch), generate a baseline CSV profile, and then diff it against the PR branch. If the P99 import time of the PR degrades by >100ms compared to the baseline, the CI check will fail.
Safety Backstop: The script still enforces an absolute hard-failure backstop of 5000ms for extreme regressions.
Conditional Execution & Graceful Skips: The pipeline verifies if a valid setup.py exists before attempting to profile, gracefully skipping directories that are not valid Python packages.
All Packages Testing: Adding the import_profile:all_packages label to a Pull Request will force the CI pipeline to run the profiler check against all packages in the mono-repo, instead of just the modified ones.

Developer Interactions

If a developer fails this CI check due to an import latency regression, they can reproduce and debug it locally by running the profiler script with the --cprofile flag (python scripts/import_profiler/profiler.py --package <their-package> --cprofile). This will generate a cProfile stack trace breakdown of the import time, allowing them to pinpoint exactly which new dependency or module initialization is causing the latency spike.

Related PRs

Builds upon the import profiler tool added in #17467

gemini-code-assist

Code Review

This pull request adds a new import_profile session to the noxfile.py.j2 template to measure and ensure import times remain below defined thresholds. Feedback points out that because this template is used in standalone repositories outside of the monorepo, referencing a relative path to the monorepo's root scripts directory will cause default nox runs to fail in those environments. It is recommended to check for the script's existence and gracefully skip the session if it is missing, while also using pathlib.Path for cleaner path handling.

…oldens

…ssing

…oro presubmit

daniel-sanche

I left a couple comments. It would be good to clarify the goal of this check though.

What counts as a failure? Can we calculate an import time diff before and after the target changes, instead of just reporting an absolute value? How are developers intended to interact with this?

daniel-sanche · 2026-07-08T22:29:54Z

+        ;;
    *)
        nox -s ${TEST_TYPE}
        retval=$?


This PR doesn't touch any packages, so it's hard to tell what the action will look like. Can you temporarily touch some files to test it out?

We should also see how long the new check would take if all packages were updated. In #17438, I added a new unit_test:all_packages tag, that will run against all packages when added. Maybe we should support that here too

It's possible we would only want to profile a few key packages instead of all of them. Most generated libraries should be very similar

I agree with Daniel on a couple of performance issues:

I would like to see this thing run so that I can understand what it will produce and whether output appears as I would expect. Please temporarily touch some files to trigger the profiler to run against them. As an example: add a blank line right after the license in this file in google-cloud-compute and a similar change in at least one other package. That will trigger the profiler to run (without requiring you to go back and undo any changes to google-cloud-compute, etc).

If you already did this and have reverted your change, please point to the commit that you used to trigger the profiler (we should be able to check the CI/CD results for that commit).

Note
In case you've never done it... clicking on the x OR checkmark next to the commit number grants access to the test results for previous commits:

If, as Daniel suggests, we run the profiler against all packages, it would be useful to know how long that CI/CD check will take. We have 240 generated packages and if we regen and profile all of them we don't wanna find out later that it takes hours. Do we have benchmark results to give us an order of magnitude for how long it takes?

I've added a temporary commit (8a2d75b) that adds a blank line to google-cloud-compute and google-api-core so you can see the profiler run on a couple of real packages and review the output.

Regarding the benchmark on all packages, we actually already support running the profiler against all packages in this PR! Similar to how the unit_test:all_packages label works, there's logic in .github/workflows/import-profiler.yml that checks for an import_profile:all_packages label.

If we want to see how long it takes across all ~240 packages to get an order of magnitude, we can simply apply the import_profile:all_packages label to this PR and let it run.

…x syntax errors

hebaalazzeh · 2026-07-09T04:14:30Z

/gemini

chalmerlowe · 2026-07-09T15:57:47Z

+        python -m pip install --upgrade setuptools pip wheel
+        python -m pip install nox


I might be missing it, if so, sorry. I don't see any specific references to your import profiler job using nox. We install it here, but when I check for the import_profile) case statement in run_single_test.sh` I don't see any nox references. Every other case explicitly uses nox to run their tests.

Note
Since you do not launch your script using a function from the noxfile.py, you should NOT need to install nox.

…king

chalmerlowe · 2026-07-09T17:01:15Z

I reviewed the results of the profiler run. It appears as though it is partially successful and yet, halfway through each run for google-cloud-compute and google-api-core an error message appears

Successfully built google-cloud-compute
Installing collected packages: google-cloud-compute
  Attempting uninstall: google-cloud-compute
    Found existing installation: google-cloud-compute 1.49.0
    Uninstalling google-cloud-compute-1.49.0:
      Successfully uninstalled google-cloud-compute-1.49.0
Successfully installed google-cloud-compute-1.49.0
usage: profiler.py [-h] [--module MODULE] [--iterations ITERATIONS]
                   [--cpu CPU] [--csv CSV] [--trace] [--cprofile] [--mprofile]
                   [--keep-pycache]
profiler.py: error: unrecognized arguments: --package google-cloud-compute
Previous HEAD position was 006c9479b chore: add reauth extra to constraints (#17685)

Later in the same stretch we see some results that indicate a failure but do so in a way that is non-intuitive:
The FAILURE line is buried in the middle of the results. Typically success and failure messages are the last OR nearly last thing to display.

CPU Pinning enabled: Pinning processes to core 0 using taskset.
--- Results for google.cloud.compute (10 iterations) ---
Code Volume (Deterministic):
  Loaded Modules: 1320
  Loaded Lines:   1023913
Time (ms):
  P50 (Median): 13919.05
  P90:          14474.85
  P99:          14981.28
  Mean:         13981.90
  Min:          13823.31
  Max:          14526.00
  StdDev:       199.01

Tracemalloc RAM (MB):
  P50 (Median): 115.4449
  P90:          115.4457
  P99:          115.4461
  Mean:         115.4448
  Min:          115.4435
  Max:          115.4457
  StdDev:       0.0006

FAILURE: P99 import time (14981.28 ms) exceeds the failure threshold (5000.0 ms).
Physical RSS RAM (MB):
  P50 (Median): 227.4062
  P90:          227.4719
  P99:          227.4796
  Mean:         227.4008
  Min:          227.3086
  Max:          227.4727
  StdDev:       0.0532

If the built in profiler injects that line in its results, there may be nothing we can do about it, none-the-less, what we want to happen is for our process in its entirety to complete with either a success flag OR fail flag and have a message at the end.

Session import_profiler was successful in 2 minutes.
Session import_profiler was failed in 3 minutes.

Something like this at the end of your profiler script could do the trick of signaling to the CI/CD workflow that the script resulted in a success OR failure result:

    # Distinct exit codes for CI/CD
    if <status is success>:
        sys.exit(0)
    else:
        sys.exit(1)

hebaalazzeh added 2 commits July 8, 2026 02:36

ci: add import profiler check across monorepo

6d60f59

ci: allow prerelease for python 3.15 in setup-python

da118aa

gemini-code-assist Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread packages/gapic-generator/gapic/templates/noxfile.py.j2 Outdated

hebaalazzeh added 7 commits July 8, 2026 02:52

chore: propagate import_profile nox session across all packages and g…

5eab435

…oldens

fix(ci): gracefully skip import_profile session if profiler script mi…

8ccfb28

…ssing

style(ci): reformat noxfile with black to fix lint errors

a0c075f

chore: revert google-crc32c noxfile modification to bypass broken kok…

a7d2363

…oro presubmit

fix(ci): fix bazel integration tests and lint formatting

68a0bdf

chore: revert gapic-generator template and noxfile changes in packages

42dd64a

ci: run import_profile without nox templates

a6491b5

daniel-sanche reviewed Jul 8, 2026

View reviewed changes

address PR comments for import profiler CI

9472ab6

hebaalazzeh force-pushed the feat/import-profiler-ci branch from 113b3d5 to 9472ab6 Compare July 8, 2026 23:50

hebaalazzeh added 3 commits July 9, 2026 02:39

fix: update goldens for gapic-generator integration tests

705b188

feat: implement dynamic import profile diff check against baseline

885de7f

refactor: simplify CI without gapic-generator template changes and fi…

a2bd6ec

…x syntax errors

hebaalazzeh marked this pull request as ready for review July 9, 2026 04:09

hebaalazzeh requested a review from a team as a code owner July 9, 2026 04:09

chalmerlowe reviewed Jul 9, 2026

View reviewed changes

hebaalazzeh added 3 commits July 9, 2026 16:19

chore: remove nox installation from import profiler job

a77dc18

chore: temporary commit to trigger profiler

8a2d75b

chore: temporarily force profiler to run on all packages for benchmar…

9732c4e

…king

fix: resolve profiler baseline args and improve failure logs

df592fe

		python -m pip install --upgrade setuptools pip wheel
		python -m pip install nox

Uh oh!

Conversation

hebaalazzeh commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Changes Included

Developer Interactions

Related PRs

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

daniel-sanche left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

daniel-sanche Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

hebaalazzeh Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hebaalazzeh Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hebaalazzeh commented Jul 9, 2026

Uh oh!

chalmerlowe Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

hebaalazzeh Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hebaalazzeh commented Jul 8, 2026 •

edited

Loading

daniel-sanche left a comment •

edited

Loading

chalmerlowe Jul 9, 2026 •

edited

Loading